Fix flaky test:Move method invocation counter increment to call() method#3274
Conversation
📝 WalkthroughWalkthroughInvokeMethodRunnable: runOne() centralizes Throwable handling and wraps failures as TestNGRuntimeException; call() moves m_method.incrementCurrentInvocationCount() into a finally so the invocation count increments on all exit paths. ChangesException Handling and Counter Management
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java (1)
70-85: ⚡ Quick winAdd a regression test and a CHANGES.txt entry for this flaky-test fix.
The PR description acknowledges that tests and
CHANGES.txtwere not updated. For a fix targeted at a flaky test (issue#2586, related to#599), a deterministic regression test that exercises the timeout/invocationCountpath and asserts exactly one increment ofgetCurrentInvocationCount()(and one passed result inverifyTests()) would prevent future regressions of the same class of bug — particularly given that the counter-increment semantics under timeouts are now subtly different from the previous behavior. A short note inCHANGES.txtreferencing#2586is also worth adding for downstream consumers.Want me to draft a regression test (e.g. a small
HookableTest-style class withinvocationCount+invocationTimeOutplus a verifier ongetCurrentInvocationCount()), and/or aCHANGES.txtline referencing#2586?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java` around lines 70 - 85, Add a deterministic regression test and a CHANGES.txt entry to prevent regressions around the timeout+invocationCount behavior fixed in InvokeMethodRunnable.call(): create a small HookableTest-like test that sets a method with invocationCount > 1 and invocationTimeOut > 0, exercise the timeout path so only one run completes, assert that m_method.getCurrentInvocationCount() == 1 (and that verifyTests() reports one passed result), and ensure the test reliably reproduces the previous flaky behavior; also add a short CHANGES.txt line referencing issue `#2586` describing the flaky-test fix so downstream consumers are aware.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java`:
- Around line 63-66: The outer catch in InvokeMethodRunnable.runOne that catches
Exception and rethrows new TestNGRuntimeException(e) is redundant and
double-wraps an existing TestNGRuntimeException produced by the inner
catch(Throwable) block; remove that outer try/catch so the inner logic (which
computes the underlying cause and throws the original TestNGRuntimeException
stored/rethrown as t) is the single propagation path, preserving the original
exception identity and cause.
---
Nitpick comments:
In
`@testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java`:
- Around line 70-85: Add a deterministic regression test and a CHANGES.txt entry
to prevent regressions around the timeout+invocationCount behavior fixed in
InvokeMethodRunnable.call(): create a small HookableTest-like test that sets a
method with invocationCount > 1 and invocationTimeOut > 0, exercise the timeout
path so only one run completes, assert that m_method.getCurrentInvocationCount()
== 1 (and that verifyTests() reports one passed result), and ensure the test
reliably reproduces the previous flaky behavior; also add a short CHANGES.txt
line referencing issue `#2586` describing the flaky-test fix so downstream
consumers are aware.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3b339a41-2474-4b2f-9dd8-03e0d31d2255
📒 Files selected for processing (1)
testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java
Closes testng-team#2586 This commit fixes the flaky test in test.hook.HookableTest where tests with timeouts (related to issue testng-team#599) would inconsistently report pass counts. Changes: - Move incrementCurrentInvocationCount() from runOne() to call() method's finally block to ensure it's called exactly once per method execution - Add try/finally block in call() to guarantee counter increment even when exceptions occur - Improve exception handling in runOne() to properly propagate errors - Remove multiple invocation counting when tests are run with timeouts The root cause was that the invocation counter could be incremented multiple times or inconsistently during test execution with timeouts, leading to flaky behavior where verifyTests() would fail with "expected [1] but found [0]". This change ensures reliable test counting regardless of timeout settings or execution conditions, fixing issue testng-team#2586.
1bb4eff to
35d8711
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java (1)
52-59:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAvoid re-wrapping
TestNGRuntimeExceptioninrun()
runOne()now throwsTestNGRuntimeException, butrun()catchesExceptionand wraps it again, producing nestedTestNGRuntimeExceptionand obscuring the root cause chain.Proposed fix
public boolean run() throws TestNGRuntimeException { try { return call(); + } catch (TestNGRuntimeException e) { + throw e; } catch (Exception e) { throw new TestNGRuntimeException(e); } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java` around lines 52 - 59, The catch in InvokeMethodRunnable.run() re-wraps exceptions from runOne() into another TestNGRuntimeException; update the catch logic so that if the caught Throwable (or its cause) is already a TestNGRuntimeException you rethrow it as-is instead of creating a new wrapper: inspect e (and e.getCause()), if instanceof TestNGRuntimeException assign that instance to t (or throw it directly), otherwise wrap the root cause in a new TestNGRuntimeException as currently done; ensure this change is applied in the catch block that sets invoked and t so run() does not produce nested TestNGRuntimeException instances.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In
`@testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java`:
- Around line 52-59: The catch in InvokeMethodRunnable.run() re-wraps exceptions
from runOne() into another TestNGRuntimeException; update the catch logic so
that if the caught Throwable (or its cause) is already a TestNGRuntimeException
you rethrow it as-is instead of creating a new wrapper: inspect e (and
e.getCause()), if instanceof TestNGRuntimeException assign that instance to t
(or throw it directly), otherwise wrap the root cause in a new
TestNGRuntimeException as currently done; ensure this change is applied in the
catch block that sets invoked and t so run() does not produce nested
TestNGRuntimeException instances.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ce5416e9-11a4-42e3-ad1a-d91016e9279e
📒 Files selected for processing (1)
testng-core/src/main/java/org/testng/internal/invokers/InvokeMethodRunnable.java
Closes #2586
This commit fixes the flaky test in test.hook.HookableTest where tests with timeouts (related to issue #599) would inconsistently report pass counts.
Changes:
The root cause was that the invocation counter could be incremented multiple times or inconsistently during test execution with timeouts, leading to flaky behavior where verifyTests() would fail with "expected [1] but found [0]".
This change ensures reliable test counting regardless of timeout settings or execution conditions, fixing issue #2586.
Fixes #2586 .
Did you remember to?
CHANGES.txt./gradlew autostyleApplyWe encourage pull requests that:
If your pull request involves fixing SonarQube issues then we would suggest that you please discuss this with the
TestNG-dev before you spend time working on it.
Note: For more information on contribution guidelines please make sure you refer our Contributing section for detailed set of steps.
Summary by CodeRabbit